<html>
<head><meta charset="utf-8"><title>crater outage · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20outage.html">crater outage</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="245437797"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20outage/near/245437797" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20outage.html#245437797">(Jul 09 2021 at 12:46)</a>:</h4>
<p>Crater seems to have been down over the last ~12 hours. I'm still not really sure why, but it seems like there was heavy i/o on the rust-bots server, so much so that we ran up against the 300 IOPS limit for the last 11 hours. I changed the disk type on the instance from gp2 to gp3 to bump the iops "for free" and that has cleared the backlog seemingly, so we seem to be back up now.</p>



<a name="245440085"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20outage/near/245440085" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aaron Hill <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20outage.html#245440085">(Jul 09 2021 at 13:07)</a>:</h4>
<p>My crater run had ~25k regressions, so that might have been related</p>



<a name="245440421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20outage/near/245440421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20outage.html#245440421">(Jul 09 2021 at 13:10)</a>:</h4>
<p>Yeah, it's possible</p>



<a name="245467829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20outage/near/245467829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20outage.html#245467829">(Jul 09 2021 at 16:51)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> gp3 definitely seems like the right answer.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>